Skip to content

fix: add per-address rate limiting and reject-when-busy for gen_call - #1443

Merged
MuncleUscles merged 1 commit into
mainfrom
fix/gen-call-reject-when-busy
Feb 13, 2026
Merged

MuncleUscles merged 1 commit into
mainfrom
fix/gen-call-reject-when-busy

Conversation

@MuncleUscles

@MuncleUscles MuncleUscles commented Feb 13, 2026 •

Copy link
Copy Markdown
Member

Summary

Emergency fix for Studio PRD crash loop — a single contract hammering gen_call exhausts the DB connection pool (QueuePool 150 limit), causing both jsonrpc pods to crash-loop with 502s.

Two protection layers:

  • Per-address rate limit: sliding window (20 req/10s per contract address). Rejects early with -32005 before acquiring DB connections or GenVM slots
  • Non-blocking semaphore: when all GenVM execution slots are occupied, rejects immediately with -32006 instead of queuing (queued requests hold DB connections while waiting)

Both errors include retry_after_seconds for client backoff.

Observability

  • Rate limit rejections logged at WARNING with contract address
  • Capacity rejections logged at WARNING with slot count
  • New /health/ratelimit endpoint shows per-address request counts, semaphore availability

Config (env vars)

Var Default Description
GEN_CALL_RATE_LIMIT_WINDOW 10 Window size in seconds
GEN_CALL_RATE_LIMIT_MAX 20 Max requests per window per address
GENVM_MAX_CONCURRENT 8 Max concurrent GenVM executions (existing)

Test plan

  • Existing integration tests pass (gen_call endpoint still works)
  • Verify rate limit rejects after threshold
  • Verify capacity rejection when semaphore full
  • Monitor /health/ratelimit after deploy

Summary by CodeRabbit

Release Notes

  • New Features
    • Introduced per-address rate limiting for RPC calls with sliding-window enforcement and retry guidance in error responses.
    • Added server capacity checks to prevent overload by rejecting requests when maximum concurrent operations reached.
    • New diagnostic endpoint to monitor rate-limit status and per-address request metrics.

…_call

Prevents a single contract from exhausting the DB connection pool by
hammering gen_call. Two layers:

1. Per-address sliding window rate limit (20 req/10s default) — rejects
   before any heavy resources are acquired
2. Non-blocking semaphore check — rejects immediately when all GenVM
   execution slots are occupied instead of queuing and holding DB connections

Both return specific JSON-RPC error codes (-32005 rate limit, -32006
server busy) with retry_after_seconds so clients can back off.

Adds /health/ratelimit endpoint for observability.

Env vars: GEN_CALL_RATE_LIMIT_WINDOW, GEN_CALL_RATE_LIMIT_MAX, GENVM_MAX_CONCURRENT
@coderabbitai

coderabbitai Bot commented Feb 13, 2026 •

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The changes introduce per-address rate limiting for gen_call/sim_call endpoints using a sliding-window mechanism and add a diagnostic health endpoint to monitor rate limit state and GenVM capacity metrics across active addresses.

Changes

Cohort / File(s) Summary
Rate Limiting Implementation
backend/protocol_rpc/endpoints.py
Adds sliding-window rate limiting with per-address request tracking. Implements _check_rate_limit() function that maintains timestamps and enforces configurable limits, raising JSONRPCError(-32005) when exceeded. Also adds global GenVM semaphore capacity check raising JSONRPCError(-32006) when busy. Integrates rate limit checks early in validation flow before resource allocation.
Health Monitoring
backend/protocol_rpc/health.py
Introduces /health/ratelimit GET endpoint exposing rate limit configuration, semaphore availability, active addresses count, and per-address metrics including requests within window, limit threshold, and oldest request age. Provides read-only diagnostic visibility into current rate limiting state.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant Validator as Validator
    participant RateLimit as Rate Limiter
    participant GenVM as GenVM
    participant Log as Request Log

    Client->>Validator: gen_call request
    Validator->>RateLimit: _check_rate_limit(address)
    RateLimit->>Log: query recent requests (sliding window)
    Log-->>RateLimit: timestamps within window
    
    alt Within limit
        RateLimit-->>Validator: allow
        Validator->>GenVM: acquire semaphore
        alt Semaphore available
            GenVM-->>Validator: proceed
            Validator-->>Client: execute call
        else Semaphore locked
            GenVM-->>Validator: JSONRPCError(-32006)
            Validator-->>Client: server busy
        end
    else Exceeded limit
        RateLimit-->>Validator: JSONRPCError(-32005)
        Validator-->>Client: rate limit exceeded
    end
    
    Validator->>Log: record request timestamp
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

  • #1439: Adds GenVM concurrency limiting and metrics, which this PR builds upon with per-address rate limiting in the same endpoint file.

Suggested labels

run-tests

Poem

🐰 Hopping through the token stream,
Rate limits keep things lean and keen,
A sliding window, oh so fair,
Bounds each address with gentle care—
No flood here, just steady pace,
GenVM smiles with breathing space! ✨

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main changes: adding per-address rate limiting and reject-when-busy functionality for gen_call endpoint.
Description check ✅ Passed The description covers all key sections: summary of the emergency fix, two protection layers explained, observability features, configuration variables, and a test plan.
Docstring Coverage ✅ Passed Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Merge Conflict Detection ✅ Passed ✅ No merge conflicts detected when merging into main

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing touches
  • 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment
  • Commit unit tests in branch fix/gen-call-reject-when-busy

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Fix all issues with AI agents
In `@backend/protocol_rpc/endpoints.py`:
- Around line 77-98: The _address_request_log dictionary accumulates stale
entries for addresses that never get checked again; modify _check_rate_limit to
perform amortised cleanup by pruning expired timestamps for all addresses as
well as the current one: compute cutoff = now - _RATE_LIMIT_WINDOW, iterate over
keys of _address_request_log and replace each list with only timestamps >
cutoff, and delete any address whose list becomes empty to free memory; then
proceed with the existing per-address rate check logic (using
_RATE_LIMIT_MAX/_RATE_LIMIT_WINDOW) and keep logging via _rate_limit_logger as
before.

In `@backend/protocol_rpc/health.py`:
- Line 941: Remove the unused "noqa: SLF001" and stop reading the CPython
internal attribute _genvm_semaphore._value; instead expose availability via a
public API or compute it from a tracked counter. Replace the direct access to
_genvm_semaphore._value in the health payload by calling a new/existing public
accessor (e.g., genvm_semaphore.available() or get_genvm_semaphore_available())
or compute availability as (concurrency_limit - genvm_in_use_counter) maintained
where acquire/release happen; update the code paths that change the semaphore to
increment/decrement that counter so the health endpoint can report availability
without touching internal attributes.
🧹 Nitpick comments (3)
backend/protocol_rpc/endpoints.py (1)

974-975: Rate-limit check happens after validator snapshot acquisition, not "early".

By the time _check_rate_limit(to_address) runs here, _execute_call_with_snapshot has already created the validator snapshot (line 875) which involves DB work. If the goal is to reject before heavyweight resource allocation, consider moving the check into _execute_call_with_snapshot right after extracting to_address from params["to"], before the snapshot context manager.

backend/protocol_rpc/health.py (2)

924-924: Redundant import time as _time — time is already imported at module level.

Suggested fix
-    import time as _time
-
-    now = _time.monotonic()
+    now = time.monotonic()

929-936: Iterating _address_request_log without pruning stale entries from the shared dict.

This endpoint reads the log but doesn't clean up fully-expired addresses (same issue as in endpoints.py). For a diagnostic endpoint this is acceptable, but be aware that the addresses payload can grow large over time if many unique contracts have been rate-limited.

Comment on lines +77 to +98
_address_request_log: dict[str, list[float]] = {} # {address: [timestamp, ...]}

_rate_limit_logger = logging.getLogger(__name__ + ".rate_limit")


def _check_rate_limit(address: str) -> None:
"""Reject if address exceeds rate limit. Prunes old entries."""
now = time.monotonic()
timestamps = _address_request_log.get(address, [])
cutoff = now - _RATE_LIMIT_WINDOW
timestamps = [t for t in timestamps if t > cutoff]
if len(timestamps) >= _RATE_LIMIT_MAX:
_rate_limit_logger.warning(
f"Rate limit exceeded for {address}: {len(timestamps)} requests in {_RATE_LIMIT_WINDOW}s window"
)
raise JSONRPCError(
code=-32005,
message=f"Rate limit exceeded: max {_RATE_LIMIT_MAX} gen_call requests per {_RATE_LIMIT_WINDOW}s per contract address",
data={"address": address, "retry_after_seconds": _RATE_LIMIT_WINDOW},
)
timestamps.append(now)
_address_request_log[address] = timestamps

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

_address_request_log grows without bound for inactive addresses.

Entries are only pruned when the same address is checked again. Addresses that go idle leave stale entries in the dict forever. Over time in production (many distinct contract addresses), this is a slow memory leak.

Consider periodically evicting addresses whose timestamps have all expired — for example, by cleaning during each _check_rate_limit call (amortised), or via a background task.

♻️ Lightweight amortised cleanup sketch
+import itertools
+
+_CLEANUP_EVERY = 500  # prune stale addresses every N calls
+_check_count = 0
+
 def _check_rate_limit(address: str) -> None:
     """Reject if address exceeds rate limit. Prunes old entries."""
+    global _check_count
     now = time.monotonic()
+    cutoff = now - _RATE_LIMIT_WINDOW
+
+    # Amortised full sweep
+    _check_count += 1
+    if _check_count >= _CLEANUP_EVERY:
+        _check_count = 0
+        stale = [a for a, ts in _address_request_log.items() if ts and ts[-1] <= cutoff]
+        for a in stale:
+            del _address_request_log[a]
+
     timestamps = _address_request_log.get(address, [])
-    cutoff = now - _RATE_LIMIT_WINDOW
     timestamps = [t for t in timestamps if t > cutoff]
🤖 Prompt for AI Agents
In `@backend/protocol_rpc/endpoints.py` around lines 77 - 98, The
_address_request_log dictionary accumulates stale entries for addresses that
never get checked again; modify _check_rate_limit to perform amortised cleanup
by pruning expired timestamps for all addresses as well as the current one:
compute cutoff = now - _RATE_LIMIT_WINDOW, iterate over keys of
_address_request_log and replace each list with only timestamps > cutoff, and
delete any address whose list becomes empty to free memory; then proceed with
the existing per-address rate check logic (using
_RATE_LIMIT_MAX/_RATE_LIMIT_WINDOW) and keep logging via _rate_limit_logger as
before.

"window_seconds": _RATE_LIMIT_WINDOW,
"max_per_window": _RATE_LIMIT_MAX,
"genvm_concurrency_limit": _GENVM_CONCURRENCY,
"genvm_semaphore_available": _genvm_semaphore._value, # noqa: SLF001

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

🧩 Analysis chain

🌐 Web query:

asyncio.Semaphore._value Python 3.12 public API stable documentation

💡 Result:

asyncio.Semaphore._value is not a public, stable API in Python 3.12.

  • The official Python 3.12 documentation for asyncio.Semaphore documents the constructor and the methods acquire(), locked(), and release()—but does not document any .value/._value attribute as supported API. [1]
  • In CPython’s 3.12 implementation, the internal counter is stored in an instance attribute named self._value, i.e., an internal (underscore-prefixed) implementation detail that can change without notice. [2]

If you need supported behavior, rely on acquire()/release()/locked() (or maintain your own counter / wrap the semaphore), rather than reading or writing ._value. [1][2]

Sources: [1] [2]


🏁 Script executed:

# Check linting configuration to see if SLF001 is enabled
find . -name "*.toml" -o -name "*.cfg" -o -name ".ruff*" | head -20

Repository: genlayerlabs/genlayer-studio

Length of output: 83


🏁 Script executed:

# Examine the actual file and surrounding context
head -950 backend/protocol_rpc/health.py | tail -50

Repository: genlayerlabs/genlayer-studio

Length of output: 1753


🏁 Script executed:

# Check imports at the top of the file
head -30 backend/protocol_rpc/health.py

Repository: genlayerlabs/genlayer-studio

Length of output: 1088


🏁 Script executed:

# Check ruff.toml to see if SLF001 is enabled or disabled
cat ruff.toml

Repository: genlayerlabs/genlayer-studio

Length of output: 978


Remove unused noqa directive and avoid accessing asyncio.Semaphore._value, a CPython implementation detail.

The _value attribute is not part of the public API and is not documented in Python 3.12 documentation. It could break on alternative runtimes or future CPython versions. The noqa: SLF001 directive is also unused since the SLF rule is not enabled in the project's ruff.toml.

Fix: remove unused noqa
-        "genvm_semaphore_available": _genvm_semaphore._value,  # noqa: SLF001
+        "genvm_semaphore_available": _genvm_semaphore._value,

For a more robust solution, consider computing availability from the concurrency limit minus an in-use counter, or exposing it from the semaphore wrapper.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"genvm_semaphore_available": _genvm_semaphore._value, # noqa: SLF001
"genvm_semaphore_available": _genvm_semaphore._value,
🧰 Tools
🪛 Ruff (0.15.0)

[warning] 941-941: Unused noqa directive (non-enabled: SLF001)

Remove unused noqa directive

(RUF100)

🤖 Prompt for AI Agents
In `@backend/protocol_rpc/health.py` at line 941, Remove the unused "noqa: SLF001"
and stop reading the CPython internal attribute _genvm_semaphore._value; instead
expose availability via a public API or compute it from a tracked counter.
Replace the direct access to _genvm_semaphore._value in the health payload by
calling a new/existing public accessor (e.g., genvm_semaphore.available() or
get_genvm_semaphore_available()) or compute availability as (concurrency_limit -
genvm_in_use_counter) maintained where acquire/release happen; update the code
paths that change the semaphore to increment/decrement that counter so the
health endpoint can report availability without touching internal attributes.

@MuncleUscles
MuncleUscles merged commit 9f44e9c into main Feb 13, 2026
22 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 0.100.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant